Skip to content

Comments

feat: Consume SQL hook lineage in OpenLineage#62171

Merged
mobuchowski merged 2 commits intoapache:mainfrom
kacpermuda:feat-ol-sql-hll
Feb 24, 2026
Merged

feat: Consume SQL hook lineage in OpenLineage#62171
mobuchowski merged 2 commits intoapache:mainfrom
kacpermuda:feat-ol-sql-hll

Conversation

@kacpermuda
Copy link
Contributor

Followup to #61535, this PR updates the OpenLineage listener to include SQL lineage reported by hooks.

For each SQL query, it parses inputs/outputs via the existing SQLParser, enriches facets (SQL job, external query), and emits dedicated START/COMPLETE (or FAIL) child OpenLineage events. When exactly one query was executed, the parsed metadata is also merged into the parent task's lineage event, for simplicity (but the separate events are always emitted).

Updated documentation template and Sphinx extension to better describe hook-level lineage capabilities and discover these new sql hook lineage hooks.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Cursor following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Copy link
Contributor

@mobuchowski mobuchowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some nits, looks good anyway - but a question about the single/multiple events

@potiuk
Copy link
Member

potiuk commented Feb 24, 2026

I'd love @mobuchowski does the final review before I start new release (today?)

Copy link
Contributor

@mobuchowski mobuchowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One issue, good otherwise

@mobuchowski mobuchowski self-requested a review February 24, 2026 17:51
@mobuchowski mobuchowski merged commit 27426e4 into apache:main Feb 24, 2026
125 checks passed
@kacpermuda kacpermuda deleted the feat-ol-sql-hll branch February 24, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants